Bridging the Gap between the Theory and Practice of Analysis of Data from Complex Surveys - Some Statistics Canada Experiences

So as not to publish misleading results, subject matter analysts at Statistics Canada are urged to take account of the complexities of the survey design when doing analysis using data from Statistics Canada’s surveys. While commercial software packages that incorporate methods for controlling for features of the sample design are becoming more readily available and more efficient to use, analysts without some background in survey theory still have difficulty in knowing how to proceed. Statistics Canada has a small unit called the Data Analysis Resource Centre (DARC) whose purpose is to provide specialized services in analysis of statistical data. One of the major activities of DARC is the support of subject matter analysts who are using data from surveys with complex designs. This paper will present some of our experiences in DARC with assisting analysts in doing their research and some “tips and traps” that we have identified. Because of the practice in many publications, due to space restrictions, of presenting descriptive estimates without their corresponding variance estimates, new analysts of survey data are frequently surprised that they require more data about the survey design than just the final weights in order to produce acceptable variance estimates. These analysts, who are generally secondary users of survey data rather than having been involved in the implementation of the survey, welcome assistance with possible approaches to accounting for the actual survey design and tips on different software packages that can implement the approach that best suits their needs. In order to communicate our advice on these topics, DARC has found it useful to develop illustrations of real analysis using real data from the specific surveys being analysed by a group of researchers particularly the new Statistics Canada longitudinal surveys on health, youth, and labour and income dynamics. Each illustration makes use of many different software packages so that the traps and advantages of each can be pointed out. Through these illustrations, appropriate methods of variance estimation can be introduced and discussed. DARC staff members have found that this direct provision of support to analysts is also an excellent forum for identifying methodological problems such as ideosyncracies in a survey design that could be modified in the future, limitations in software, and analytical questions still requiring theoretical research.